In the film and television industry, several different standards for frame systems exist in which it is defined how motional processes have to be recorded on video or film by means of a necessarily discrete number of single frames. There is a permanent demand to reproduce image material of a special frame system by means of a projection system which is destined for another frame standard. This image material, therefore, first has to be prepared for the frame system of the projection or reproduction, respectively. This process is called image conversion or image transfer, respectively.
The present invention relates to conversion of television, computer or film images originated in a specific system of recording (i.e., a specific number of frames per second) to a system of projection designed for another system of recording (having a different number of frames per second). In this application, these different systems of recording will be referred to as "frame systems."
There are six main frame systems. They are as follows:
1. NTSC TV--60 frames/sec ("American" standard); PA1 2. PAL TV--50 frames/sec ("European" standard); PA1 3. Film 16--32 frames/sec (Archive films); PA1 4. Film 24--48 frames/sec (standard of motion-pictures); PA1 5. Film 60--120 frames/sec ("Showscan" system); and PA1 6. Computer images--Programmable number of frames/sec.
Each of these frame systems will be discussed in more detail below.
1. NTSC Television--NTSC has nominally 30 "frames" per second. However, every frame includes two separate half images called "fields." These two fields construct an NTSC video frame in such manner that the first field supplies the image information for all lines with an odd number and the second field supplies the image information for all other lines, i.e., those with even numbers. NTSC video frames therefore are called interlaced frames.
Therefore, because every field represents a different time interval, each field must be considered as a separate image. Thus, it should be considered that NTSC has 60 frames per second (fps).
2. PAL Television--PAL has nominally 25 "frames" per second. As with NTSC, however, every frame includes two separate images called "fields," which are interlaced images. Again, because every field represents a different time interval, each field must be considered as a separate image. Thus, it should be considered that PAL has 50 fps.
3. Film 16--Films produced before the introduction of sound were photographed with 16 "frames" per second. These frames can literally be seen when examining a strip of an archive film. However, during film recording and projection, there were actually 32 changes of the image per second, but only 16 of these changes represented images (i.e., "image frames"). The other 16 changes produced a black image which was the result of a closed shutter during the transportation of the film strip (i.e., "black frames"). They can be recognized on the film material as being unexposed strips between the exposed frames.
In the analysis of the motion effect in film, the importance of these invisible black images is equal to the importance of the visible images. Therefore, it should be considered that Film 16 has 32 fps.
4. Film 24--Film 24 has nominally 24 "frames" per second. These frames also can literally be seen when examining a strip of the film. As with Film 16, however, during film recording and projection there is actually twice this number of changes of the image per second, i.e., here 48 changes. Only 24 of these changes represent images (i.e., "image frames"). The other 24 changes produce a black image which was the result of a closed shutter during the transportation of the film strip (i.e., "black frames"). Thus, it should be considered that Film 24 has 48 fps.
5. Film 60--Showscan film has nominally 60 "frames" per second. These frames also can literally be seen when examining a strip of the film. There are 120 changes of the image per second during film recording and projection, but only 60 of these changes represent images (i.e., "image frames"). The other 60 changes produce a black image which was the result of a closed shutter during the transportation of the film strip (i.e., black frames"). Thus, it should be considered that Film 60 has 120 fps.
6. Computer Images--Computer images are artificially generated images which do not "photograph" reality. Movement (animation) is a calculated process which can be designed according to any of the frame systems. If computer images were designed for a particular frame system and had to be transferred to another frame system, they would need to be treated as another individual frame system. High Definition Television (television of the future) is an upgrade of NTSC and PAL standards and operates with a precise number of fps (precisely 60 fps and 50 fps).
NTSC and PAL have a frame rate slightly differing from 60 or 50 frames per second, respectively. It is, however, important for the image conversion between two systems that 60 NTSC frames and 50 PAL frames take exactly the same time, namely, approximately one second (exactly 1.0010 secs.).
As long as recorded images are projected or displayed, respectively, based on the frame system in which they were originally recorded, the effect of motion on the screen, in every case, is more or less satisfactory.
In recording original images, the best results of reproduction of motion occur in NTSC and Film 60 because these frame systems have the highest number of frames per second. The higher number of fps, the better the human eye perceives the effect of motion, i.e., the motion is smooth.
The worst original frame system for recording motion occurs in Film 16. In the case of Film 16, however, the real motion effect of the originally recorded films (at 32 fps) cannot be seen because the original means of image projection no longer exists. The "speeded up" effect of archive films is the result of modern film projection at 48 fps (FILM-24). This invention provides an opportunity to "restore" the original motion to these films.
Two types of distortion appear during film and television projections. One type of distortion is called "strobe-flicker", and the second type of distortion is called "jitter." Strobe-flicker is solely a consequence of a frame rate being too low for the creation of a flowing motion. Because of the different basic technologies of film and video on film screens or television monitors, respectively, different kinds of strobe flicker appear. This kind of disturbance, however, is in no way related to the effects which origin in the conversion of images from one frame system into another frame system.
Image jitter, on the contrary, arises from the currently applied--purely mechanical--methods of image conversions: the systematic leaving out or repetition of single images of the original system in the target system causes interruptions of the recorded motion. Since this disturbance is repeated in dependence of the frame rate in shorter or longer periods, the impression of a jittery image is caused.
U.S. Patent Nos. 1,815,455, (Waller) and 5,153,620 (Songer) try to eliminate strobe-flicker. Though, as mentioned above, strobe-flicker is not caused by image conversion, a short discussion of both patents seems to be necessary since they describe methods which are just apparently related to those of the present invention.
Waller's process seeks to reduce strobe-flicker by representing the immediate past several film frames and the immediate future several film frames on one film frame. This super-imposition results in an interesting visual effect--a "fan" of motion is built into each frame. However, this "fan" effect does, as easily can be seen, not solve the strobe-flicker or jitter problems. The Songer patent discloses superimposing two frames in one frame at an equal rate in an attempt to reduce strobe-flicker in film projection. Such superimposition results in blurred images and does not solve the strobe-flicker or jitter problems.
The methods according to Waller and Songer are not directed to reproduce the produced image material within another frame system, since the number of frames per time unit is not altered. Double or multiple exposure at constant frame rate cannot solve the image conversion problem, and furthermore, this method leads to a deterioration (blurring) of the produced image material as compared to the original material.
Today, on every frame system, images are projected which were originally recorded in other frame systems. If the original number of frames are simply mechanically projected in the new frame system (as in the case of archive Film 16), serious disturbances in the reproduction of the motion and sound would occur. For example, projection of video originally recorded in NTSC on a PAL system would appear slowed down, and projection of video originally recorded in PAL on a NTSC system would appear speeded up. To preserve the original effect of the recording, it is necessary to change the number of frames of the recorded material to the number of frames required by the system on which this recorded material is projected. This process of frame number adjustment is known as "transfer".
The known methods for transferring images from one frame system to another frame system cause jitter. The reasons therefor are explained in more detail below.
When viewing an object with the eye, normal people see one image at a time. This image is in constant motion. However, technology does not exist to record this type of image. Rather, motion is recorded as a stream of separate still frames.
In order to achieve the fluid effect of motion on the screen, an uninterrupted stream of preferably at least 60 separate still images must be observed to create an illusion of constant motion. This is the principle for every frame system. Any interruption of the stream of images (i.e., by missing or repeated frames, or improper time presentation in a frame) causes a disturbance in the visual perception of motion. Deleting a frame causes a "hole" in the motion, and the repetition of a frame causes a "freeze" of the action. To more simply and better illustrate these and additional problems caused by mechanical transfer techniques, first the example of a clock face will be used.
The object of examination is the second hand of a clock. Assume that this second hand is filmed in a frame system which records only 1 fps, and four seconds of time are being analyzed. Every frame of recorded material will represent one tick of the second hand. When this material is projected on a system whose projection requires three frames of material during a 4 second duration (i.e., every frame is projected for 1.33 seconds), and a method of mechanical transfer is applied, one frame from the recorded material is removed. Thus, one tick of the second hand is removed. A "jump" in the second hand's motion will appear on the screen. This jump is caused by the missing frame of the removed tick.
Using this same example, if the original filming is done in a frame system which records 3 frames in 4 seconds, then every frame of recorded material represents 1.33 ticks of the second hand. If this material is projected on a system whose projection requires 4 frames of material during a 4 second duration, and a method of mechanical transfer is applied, one of the frames of the recorded material must be repeated. Thus, one tick of the second hand will have to repeated. A "freeze" in the motion of the hand will appear on the screen. This freeze is caused by the repeated tick.
In addition to the "jump" or "freeze,` in both examples, the remaining frames also represent incorrect visual information. In the case where frames are removed (the "jump"), every frame is projected for 1.33 seconds, but the action actually shown in that frame represents only 1 second. In the case where frames are repeated (the "freeze"), every frame is projected for 1 second, but the action actually shown in that frame represents 1.33 seconds.
Thus, when using mechanical transfer processes, none of the transferred images represent the correct time and motion. All disturbances caused by mechanical transfer will be referred to as "jitter". Although none of the existing frame systems operate with 1 or 1.33 fps, but rather with 32, 48, 50 or 60 fps, mechanical transfer methods in the conventional frame systems causes the same jitter effects as described above.
While several attempts to solve jitter have been made by many companies in the world, no satisfactory solution to the problem has been developed.
U.S. Pat. Nos. 3,511,567 (Dejoux) and 4,889,423 (Trumbull) relate to transferring images in the motion picture and television industries. These patents are entirely incorporated herein by reference. Because these transfers are based on cutting out or repeating frames, these processes will be referred to in this application as "mechanical transfer" processes. These patents describe methods of transfer based on the following principle: if there are too many frames in the originally recorded material, then excess frames are removed. If there are not enough frames in the originally recorded material, certain frames must be repeated to provide the necessary amount.
The method of transfer described in the Dejoux patent has been practiced in the television industry for many years. However, this method of transfer does not solve the jitter problem. In fact, the Dejoux method actually causes jitter.
Trumbull discloses a method for converting from a high frame rate motion to a low frame rate motion. This is done by superimposing some images and cutting out some images. The method of Trumbull would not solve the jitter problem because the timing of the original frame system is not maintained. Furthermore, Trumbull's method would cause strobe-flicker because the human brain can distinguish twenty-four separately projected images per second.
This problem shall be explained by means of another example:
FIG. 1 illustrates a sequence of four image frames which would be produced by cinemagraphic exposure of four frames of film to an image field consisting of a moving circular object. For convenience, time durations are calibrated in degrees, wherein 360.degree. represents a sequence of four film images corresponding to a time of 1/6 sec. As illustrated in FIG. 1, the moving circular object forms an elongated image on each frame corresponding to the motion of the object during the frame exposure interval (45.degree. or 1/8 sec.). The centers of each of the four frame images illustrated are equally spaced by a time (or angle) corresponding to the image frame rate. Since it is assumed that the object is undergoing uniform linear motion, there is an equal distance between the centers of the object in the time adjacent frames and an equal spacing between the adjacent image edges in time adjacent frames. When the frames are viewed, using the standard used in recording, a viewer perceives the object moving with uniform linear motion.
FIG. 2 illustrates the series of cinemagraphic images which would result from filming the same circular object with uniform linear motion using 60 field per second television video recording and converting to cinemagraphic film 24 using the prior technique. It may be observed from FIG. 2 that the object images in each of the four frames are elongated with respect to the images that would have been obtained by original exposure using cinemagraphic techniques, and that the images on film frame 1 and film frame 2 are overlapping, while the image on film frame 2 is substantially separated from the image on film frame 3. When this film is viewed using a Film 24 projector, the viewer perceives a slower object motion during the transition from frames 1 to frame 2 and accelerated, non-uniform object motion during the transition from film frame 2 to film frame 3, giving a perception of image "jitter".
Prior to the instant invention, all existing methods of image transfer, e.g., between NTSC and PAL, were based on mechanical methods. This means a frame is always either removed or repeated. In the newest, improved methods of mechanical transfer, all efforts are concentrated on "smoothing out" either the "hole" or the "freeze". However, these smoothing out techniques do not adequately solve the problem. Smoothing out the jitter results in a loss of focus quality, and it does not solve the problem that every frame represents incorrect time.
In order to achieve the correct time-motion relation, without causing jitter, it is necessary to create a method of transfer which produces frames in the transfer recording medium which possess the identical characteristics as the frames recorded in the original projecting system. In all cases it is necessary to create completely new frames. This invention relates to methods of transfer which preserve the original time-motion relation.